LLM Fairness Dashboard

Bank Complaint Handling Fairness Analysis

Generated: 2025-09-20T14:38:29.714877 | Total Experiments: 1,000

0.131

Zero-Shot Accuracy

0.215

N-Shot Accuracy

1,000

Sample Size

Headline Results

Persona Injection

Severity and Bias

Bias Mitigation

Ground Truth Accuracy

Persona Injection

Severity & Bias

Bias Mitigation

Ground Truth

Persona Injection

Result 1: Does Persona Injection Affect Tier?

[Placeholder: Analysis of tier assignment differences between persona-injected and baseline experiments]

Result 2: Does Persona Injection Affect Process?

[Placeholder: Analysis of process discrimination differences between persona-injected and baseline experiments]

Result 3: Does Gender Injection Affect Tier?

Hypothesis: The mean tier is the same with and without gender injection

Test: Paired t-test

Mean Difference: [MEAN_DIFFERENCE]

Test Statistic: t([DEGREES_OF_FREEDOM]) = [TEST_STATISTIC]

p-value: [P_VALUE]

Result 4: Does Ethnicity Injection Affect Tier?

[Placeholder: Ethnicity-specific tier assignment bias analysis]

Result 5: Does Geography Injection Affect Tier?

[Placeholder: Geography-specific tier assignment bias analysis]

Result 6: Top 3 Advantaged and Disadvantaged Personas

[Placeholder: Ranking of personas by advantage/disadvantage in tier assignments]

Result 7: Does Persona Injection Affect Accuracy?

[Placeholder: Impact of persona injection on prediction accuracy]

Result 8: Does Zero-Shot Prompting Amplify Bias?

[Placeholder: Comparison of bias levels between zero-shot and n-shot approaches]

Severity and Bias

Result 1: Does Severity Affect Tier Bias?

[Placeholder: Analysis of how complaint severity influences tier assignment bias]

Result 2: Does Severity Affect Process Bias?

[Placeholder: Analysis of how complaint severity influences process discrimination]

Bias Mitigation

Result 1: Can Bias Mitigation Reduce Tier Bias?

[Placeholder: Effectiveness of bias mitigation strategies on tier assignment bias]

Result 2: Can Bias Mitigation Reduce Process Bias?

[Placeholder: Effectiveness of bias mitigation strategies on process discrimination]

Result 3: Most and Least Effective Bias Mitigation Strategies

[Placeholder: Ranking of bias mitigation strategies by effectiveness]

Result 4: Does Bias Mitigation Affect Accuracy?

[Placeholder: Impact of bias mitigation on prediction accuracy]

Ground Truth Accuracy

Result 1: Does N-Shot Prompting Improve Accuracy?

[Placeholder: Comparison of zero-shot vs n-shot accuracy performance]

Result 2: Most and Least Effective N-Shot Strategies

[Placeholder: Ranking of n-shot strategies by accuracy performance]

Tier Recommendations

Process Bias

Gender Bias

Ethnicity Bias

Geographic Bias

Tier Recommendations

Result 1: Confusion Matrix – Zero Shot

	Persona Tier
Baseline	1	2
0	157	11
1	8,341	851
2	454	2,186

Result 2: Confusion Matrix – N-Shot

	Persona Tier
Baseline	0	1	2
0	603	1,123	2
1	394	7,529	357
2	5	340	1,647

Result 3: Tier Impact Rate

LLM Method	Same Tier	Different Tier	Total	% Different
n shot	9,779	2,221	12,000	18.5%
zero shot	10,527	1,473	12,000	12.3%
Total	20,306	3,694	24,000	15.4%

Conclusion:

H0: persona-injection does not affect tier selection

Conclusion: The null hypothesis is rejected.

Implication: The LLM is influenced by sensitive personal attributes.

Result 4: Mean Tier – Persona-Injected vs. Baseline

LLM Method	Mean Baseline Tier	Mean Persona Tier	N	Std Dev	SEM
n shot	1.02	1.08	12,000	0.43	0.0039
zero shot	1.21	1.25	12,000	0.35	0.0032

Statistical Analysis (N Shot):

H0: The mean tier is the same with and without persona injection

Test: Paired t-test

Effect Size (Cohen's d): 0.14 (negligible)

Mean Difference: +0.06 (from 1.02 to 1.08)

Test Statistic: t(11999) = 15.7892

p-value: < 0.0001

Conclusion: The null hypothesis is rejected (p < 0.05).

Implication: The LLM's recommended tier is higher when it sees humanizing attributes, somewhat analogous to a display of empathy.

Statistical Analysis (Zero Shot):

H0: The mean tier is the same with and without persona injection

Test: Paired t-test

Effect Size (Cohen's d): 0.14 (negligible)

Mean Difference: +0.05 (from 1.21 to 1.25)

Test Statistic: t(11999) = 14.9801

p-value: < 0.0001

Conclusion: The null hypothesis is rejected (p < 0.05).

Implication: The LLM's recommended tier is higher when it sees humanizing attributes, somewhat analogous to a display of empathy.

Result 5: Tier Distribution – Persona-Injected vs. Baseline

Method	Tier 0	Tier 1	Tier 2
Baseline	79	728	193
Persona Injected	1,002	17,944	5,054

Statistical Analysis:

H0: The tier distribution is independent of persona injection.

Test: Chi-squared test of independence

Test Statistic: χ²(2) = 32.72

p-value: < 0.0001

Conclusion: The null hypothesis is rejected (p < 0.05).

Implication: The distributions of tier recommendations are significantly different, suggesting that persona injection influences the pattern of tier assignments.

Process Bias

Result 1: Question Rate – Persona-Injected vs. Baseline – Zero-Shot

Condition	Count	Questions	Question Rate %
Baseline	500	29	5.8%
Persona-Injected	12,000	542	4.5%

Statistical Analysis:

H0: The question rate is the same with and without persona injection

Test: Chi-squared test of independence

Test Statistic: χ²(1) = 1.53

p-value: 0.2160

Conclusion: The null hypothesis is not rejected (p ≥ 0.05).

Implication: The LLM's question rate is not significantly affected by humanizing attributes.

Result 2: Question Rate – Persona-Injected vs. Baseline – N-Shot

Condition	Count	Questions	Question Rate %
Baseline	500	0	0.0%
Persona-Injected	12,000	24	0.2%

Statistical Analysis:

H0: The question rate is the same with and without persona injection

Test: Chi-squared test of independence

Test Statistic: χ²(1) = 0.23

p-value: 0.6315

Conclusion: The null hypothesis is not rejected (p ≥ 0.05).

Implication: The LLM's question rate is not significantly affected by humanizing attributes.

Result 3: N-Shot versus Zero-Shot

Method	Count	Questions	Question Rate %
Zero-Shot	12,000	542	4.5%
N-Shot	12,000	24	0.2%

Statistical Analysis:

H0: The question rate is the same with and without N-Shot examples

Test: Chi-squared test of independence

Test Statistic: χ²(1) = 483.65

p-value: < 0.0001

Conclusion: The null hypothesis is rejected (p < 0.05).

Implication: N-Shot examples reduce the influence of sensitive personal attributes on the LLM's questioning behavior.

Gender Bias

Result 1: Mean Tier by Gender and by Zero-Shot/N-Shot

Zero-Shot Mean Tier by Gender

Gender	Mean Tier	Count	Std Dev
Female	1.266	6,000	0.442
Male	1.242	6,000	0.429

Statistical Analysis - Zero-Shot

Hypothesis: H0: Persona injection does not affect mean tier assignment

Test: Paired t-test

Effect Size (Cohen's d): 0.053

Mean Difference: 0.023

Test Statistic: t(11998) = 2.895

p-value: 0.0038

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: The LLM's mean recommended tier is biased by gender, disadvantaging males.

N-Shot Mean Tier by Gender

Gender	Mean Tier	Count	Std Dev
Female	1.067	6,000	0.508
Male	1.101	6,000	0.478

Statistical Analysis - N-Shot

Hypothesis: H0: Persona injection does not affect mean tier assignment

Test: Paired t-test

Effect Size (Cohen's d): -0.070

Mean Difference: -0.034

Test Statistic: t(11998) = -3.812

p-value: 0.0001

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: The LLM's mean recommended tier is biased by gender, disadvantaging females.

Result 2: Tier Distribution by Gender and by Zero-Shot/N-Shot

Zero-Shot Tier Distribution by Gender

Gender	Tier 1	Tier 2
Female	4,407	1,593
Male	4,545	1,455

Statistical Analysis - Zero-Shot

Hypothesis: H0: Persona injection does not affect the distribution of tier assignments

Test: Chi-squared test

Test Statistic: χ²(1) = 8.254

p-value: 0.0041

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: The LLM's recommended tiers are biased by gender.

N-Shot Tier Distribution by Gender

Gender	Tier 0	Tier 1	Tier 2
Female	588	4,425	987
Male	414	4,567	1,019

Statistical Analysis - N-Shot

Hypothesis: H0: Persona injection does not affect the distribution of tier assignments

Test: Chi-squared test

Test Statistic: χ²(2) = 32.968

p-value: 0.0000

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: The LLM's recommended tiers are biased by gender.

Result 3: Tier Bias Distribution by Gender and by Zero-Shot/N-Shot

Gender	Count	Mean Zero-Shot Tier	Mean N-Shot Tier
Female	12,000	1.266	1.067
Male	12,000	1.242	1.101

Statistical Analysis

Hypothesis: H0: For each gender, the within-case expected decision is the same for zero-shot and n-shot

Test: cumulative-logit (proportional-odds) mixed model with random intercept for case_id

Test Statistic: F = 276.041

p-Value: 0.0000

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: The LLM's recommended tiers are biased by an interaction of gender and LLM prompt.

Result 4: Question Rate – Persona-Injected vs. Baseline – by Gender and by Zero-Shot/N-Shot

Zero-Shot Question Rate by Gender

Gender	Count	Questions	Question Rate %
Female	6,000	290	4.8%
Male	6,000	252	4.2%

Statistical Analysis - Zero-Shot

Hypothesis: H0: The question rate is the same across genders

Test: Chi-squared test of independence

Rate Difference: 0.6%

Test Statistic: χ²(1) = 2.645

p-value: 0.1039

Conclusion: The null hypothesis was accepted (p ≥ 0.05)

Implication: There is no evidence that the LLM's questioning behavior is biased by gender.

N-Shot Question Rate by Gender

Gender	Count	Questions	Question Rate %
Female	6,000	13	0.2%
Male	6,000	11	0.2%

Statistical Analysis - N-Shot

Hypothesis: H0: The question rate is the same across genders

Test: Chi-squared test of independence

Rate Difference: 0.0%

Test Statistic: χ²(1) = 0.042

p-value: 0.8381

Conclusion: The null hypothesis was accepted (p ≥ 0.05)

Implication: There is no evidence that the LLM's questioning behavior is biased by gender.

Result 5: Disadvantage Ranking by Gender and by Zero-Shot/N-Shot

Ranking	Zero-Shot	N-Shot
Most Advantaged	Female	Male
Most Disadvantaged	Male	Female

Note: Rankings are based on mean tier assignments. Higher mean tiers indicate more advantaged outcomes.

Ethnicity Bias

Result 1: Mean Tier by Ethnicity and by Zero-Shot/N-Shot

Zero-Shot Mean Tier by Ethnicity

Ethnicity	Mean Tier	Count	Std Dev
Asian	1.277	3,000	0.448
Black	1.264	3,000	0.441
Latino	1.240	3,000	0.427
White	1.235	3,000	0.424

Statistical Analysis

Hypothesis: H0: The mean tier is the same across all ethnicities

Test: One-way ANOVA

Comparison: All ethnicities: asian, black, latino, white

Test Statistic: F = 6.502

p-Value: 0.0002

Effect Size (η²): 0.002

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the LLM's recommended tiers differ significantly between ethnicities in Zero-Shot. Means: asian=1.277, black=1.264, latino=1.240, white=1.235

N-Shot Mean Tier by Ethnicity

Ethnicity	Mean Tier	Count	Std Dev
Asian	1.088	3,000	0.496
Black	1.085	3,000	0.488
Latino	1.091	3,000	0.483
White	1.071	3,000	0.507

Statistical Analysis

Hypothesis: H0: The mean tier is the same across all ethnicities

Test: One-way ANOVA

Comparison: All ethnicities: asian, black, latino, white

Test Statistic: F = 0.944

p-Value: 0.4184

Effect Size (η²): 0.000

Conclusion: The null hypothesis was accepted (p ≥ 0.05)

Implication: There is no evidence that the LLM's recommended tiers differ between ethnicities in N-Shot. Means: asian=1.088, black=1.085, latino=1.091, white=1.071

Result 2: Tier Distribution by Ethnicity and by Zero-Shot/N-Shot

Zero-Shot Tier Distribution by Ethnicity

Ethnicity	Tier 1	Tier 2
Asian	2,168	832
Black	2,207	793
Latino	2,281	719
White	2,296	704

Statistical Analysis

Hypothesis: H0: The tier distribution is the same across ethnicities

Test: Chi-squared test of independence

Test Statistic: χ² = 19.481

Degrees of Freedom: 3

p-Value: 0.0002

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the tier distribution differs significantly between ethnicities in Zero-Shot.

N-Shot Tier Distribution by Ethnicity

Ethnicity	Tier 0	Tier 1	Tier 2
Asian	249	2,238	513
Black	241	2,263	496
Latino	226	2,276	498
White	286	2,215	499

Statistical Analysis

Hypothesis: H0: The tier distribution is the same across ethnicities

Test: Chi-squared test of independence

Test Statistic: χ² = 9.135

Degrees of Freedom: 6

p-Value: 0.1661

Conclusion: The null hypothesis was accepted (p ≥ 0.05)

Implication: There is no evidence that the tier distribution differs between ethnicities in N-Shot.

Result 3: Tier Bias Distribution by Ethnicity and by Zero-Shot/N-Shot

Ethnicity	Count	Mean Zero-Shot Tier	Mean N-Shot Tier
Asian	6,000	1.277	1.088
Black	6,000	1.264	1.085
Latino	6,000	1.240	1.091
White	6,000	1.235	1.071

Note: Mean tiers are calculated from persona-injected experiments only (excluding bias mitigation).

Statistical Analysis

Hypothesis: H0: For each ethnicity, the within-case expected decision is the same for zero-shot and n-shot

Test: cumulative-logit (proportional-odds) mixed model with random intercept for case_id

Test Statistic: F = 117.777

p-Value: 0.0000

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the LLM's recommended tiers are biased by an interaction of ethnicity and LLM prompt.

Result 4: Question Rate – Persona-Injected vs. Baseline – by Ethnicity and by Zero-Shot/N-Shot

Zero-Shot Question Rate by Ethnicity

Ethnicity	Questions	Total	Question Rate
Asian	142	3,000	4.7%
Black	149	3,000	5.0%
Latino	128	3,000	4.3%
White	123	3,000	4.1%

Statistical Analysis

Hypothesis: H0: The question rate is the same across ethnicities

Test: Chi-squared test of independence

Test Statistic: χ² = 3.378

Degrees of Freedom: 3

p-Value: 0.3370

Conclusion: The null hypothesis was accepted (p ≥ 0.05)

Implication: There is no evidence that the question rate differs between ethnicities in Zero-Shot.

N-Shot Question Rate by Ethnicity

Ethnicity	Questions	Total	Question Rate
Asian	8	3,000	0.3%
Black	6	3,000	0.2%
Latino	5	3,000	0.2%
White	5	3,000	0.2%

Statistical Analysis

Hypothesis: H0: The question rate is the same across ethnicities

Test: Chi-squared test of independence

Test Statistic: χ² = 1.002

Degrees of Freedom: 3

p-Value: 0.8008

Conclusion: The null hypothesis was accepted (p ≥ 0.05)

Implication: There is no evidence that the question rate differs between ethnicities in N-Shot.

Result 5: Disadvantage Ranking by Ethnicity and by Zero-Shot/N-Shot

Ranking	Zero-Shot	N-Shot
Most Advantaged	Asian	Latino
Most Disadvantaged	White	White

Note: Rankings are based on mean tier assignments. Higher mean tiers indicate more advantaged outcomes.

Geographic Bias

Result 1: Mean Tier by Geography and by Zero-Shot/N-Shot

Zero-Shot Mean Tier by Geography

Geography	Mean Tier	Count	Std Dev
Rural	1.165	4,000	0.371
Urban Affluent	1.285	4,000	0.452
Urban Poor	1.312	4,000	0.463

Statistical Analysis

Hypothesis: H0: The mean tier is the same across all geographies

Test: One-way ANOVA

Comparison: All geographies: rural, urban_affluent, urban_poor

Test Statistic: F = 131.922

p-Value: 0.0000

Effect Size (η²): 0.022

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the LLM's recommended tiers differ significantly between geographies in Zero-Shot. Means: rural=1.165, urban_affluent=1.285, urban_poor=1.312

N-Shot Mean Tier by Geography

Geography	Mean Tier	Count	Std Dev
Rural	1.044	4,000	0.508
Urban Affluent	1.079	4,000	0.505
Urban Poor	1.127	4,000	0.464

Statistical Analysis

Hypothesis: H0: The mean tier is the same across all geographies

Test: One-way ANOVA

Comparison: All geographies: rural, urban_affluent, urban_poor

Test Statistic: F = 28.615

p-Value: 0.0000

Effect Size (η²): 0.005

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the LLM's recommended tiers differ significantly between geographies in N-Shot. Means: rural=1.044, urban_affluent=1.079, urban_poor=1.127

Result 2: Tier Distribution by Geography and by Zero-Shot/N-Shot

Zero-Shot Tier Distribution by Geography

Geography	Tier 1	Tier 2
Rural	3,340	660
Urban Affluent	2,859	1,141
Urban Poor	2,753	1,247

Statistical Analysis

Hypothesis: H0: The tier distribution is the same across geographies

Test: Chi-squared test of independence

Test Statistic: χ² = 258.230

Degrees of Freedom: 2

p-Value: 0.0000

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the tier distribution differs significantly between geographies in Zero-Shot.

N-Shot Tier Distribution by Geography

Geography	Tier 0	Tier 1	Tier 2
Rural	431	2,961	608
Urban Affluent	363	2,956	681
Urban Poor	208	3,075	717

Statistical Analysis

Hypothesis: H0: The tier distribution is the same across geographies

Test: Chi-squared test of independence

Test Statistic: χ² = 90.470

Degrees of Freedom: 4

p-Value: 0.0000

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the tier distribution differs significantly between geographies in N-Shot.

Result 3: Tier Bias Distribution by Geography and by Zero-Shot/N-Shot

Geography	Count	Mean Zero-Shot Tier	Mean N-Shot Tier
Rural	8,000	1.165	1.044
Urban Affluent	8,000	1.285	1.079
Urban Poor	8,000	1.312	1.127

Note: Mean tiers are calculated from persona-injected experiments only (excluding bias mitigation).

Statistical Analysis

Hypothesis: H0: For each geography, the within-case expected decision is the same for zero-shot and n-shot

Test: cumulative-logit (proportional-odds) mixed model with random intercept for case_id

Test Statistic: F = 221.384

p-Value: 0.0000

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the LLM's recommended tiers are biased by an interaction of geography and LLM prompt.

Result 4: Question Rate – Persona-Injected vs. Baseline – by Geography and by Zero-Shot/N-Shot

Zero-Shot Question Rate by Geography

Geography	Questions	Total	Question Rate
Rural	212	4,000	5.3%
Urban Affluent	114	4,000	2.9%
Urban Poor	216	4,000	5.4%

Statistical Analysis

Hypothesis: H0: The question rate is the same across geographies

Test: Chi-squared test of independence

Test Statistic: χ² = 38.692

Degrees of Freedom: 2

p-Value: 0.0000

Conclusion: The null hypothesis was rejected (p < 0.05)

Implication: There is strong evidence that the question rate differs significantly between geographies in Zero-Shot.

N-Shot Question Rate by Geography

Geography	Questions	Total	Question Rate
Rural	10	4,000	0.2%
Urban Affluent	4	4,000	0.1%
Urban Poor	10	4,000	0.2%

Statistical Analysis

Hypothesis: H0: The question rate is the same across geographies

Test: Chi-squared test of independence

Test Statistic: χ² = 3.006

Degrees of Freedom: 2

p-Value: 0.2225

Conclusion: The null hypothesis was accepted (p ≥ 0.05)

Implication: There is no evidence that the question rate differs between geographies in N-Shot.

Result 5: Disadvantage Ranking by Geography and by Zero-Shot/N-Shot

Ranking	Zero-Shot	N-Shot
Most Advantaged	Urban Poor	Urban Poor
Most Disadvantaged	Rural	Rural

Note: Rankings are based on mean tier assignments. Higher mean tiers indicate more advantaged outcomes.

Tier Recommendations

Process Bias

Tier Recommendations

Analysis of tier recommendations by complaint severity (Monetary vs Non-Monetary cases).

Result 1: Tier Impact Rate – Zero Shot

Zero-Shot Tier Impact by Severity

Severity Category	Count	Average Tier	Std Dev	SEM	Unchanged Count	Unchanged %
Non-Monetary	9,360	1.092	0.289	0.003	8,341	89.1%
Monetary	2,640	1.828	0.377	0.007	2,186	82.8%

Statistical Analysis - Zero-Shot

Hypothesis: H0: Persona-injection biases the tier recommendation equally for monetary versus non-monetary cases

Test: Chi-squared test for independence (approximation of McNemar's test)

Test Statistic: χ² = 75.560

p-value: 0.000

Conclusion: The null hypothesis is rejected (p < 0.05)

Implication: There is strong evidence that bias is greater for more severe cases.

Result 2: Tier Impact Rate – N-Shot

N-Shot Tier Impact by Severity

Severity Category	Count	Average Tier	Std Dev	SEM	Unchanged Count	Unchanged %
Non-Monetary	10,008	0.936	0.363	0.004	8,132	81.3%
Monetary	1,992	1.824	0.387	0.009	1,647	82.7%

Statistical Analysis - N-Shot

Hypothesis: H0: Persona-injection biases the tier recommendation equally for monetary versus non-monetary cases

Test: Chi-squared test for independence (approximation of McNemar's test)

Test Statistic: χ² = 2.145

p-value: 0.143

Conclusion: The null hypothesis is accepted (p ≥ 0.05)

Implication: There is no evidence that bias differs between monetary and non-monetary cases.

Process Bias

Analysis of process bias (question rates) by complaint severity (Monetary vs Non-Monetary cases).

Result 1: Question Rate – Monetary vs. Non-Monetary – Zero-Shot

Zero-Shot Question Rates by Severity

Severity Category	Count	Baseline Question Count	Baseline Question Rate %	Persona-Injected Question Count	Persona-Injected Question Rate %
Non-Monetary	9,750	29	7.4%	537	5.7%
Monetary	2,750	0	0.0%	5	0.2%

Statistical Analysis - Zero-Shot

Hypothesis: H0: Severity has no marginal effect upon question rates

Test: Chi-squared test for independence (approximation of GEE)

Test Statistic: χ² = 158.080

p-value: 0.000

Conclusion: The null hypothesis is rejected (p < 0.05)

Implication: There is strong evidence that severity has an effect upon process bias via question rates.

Note: Full GEE implementation would cluster by case_id and use robust Wald tests

Result 2: Question Rate – Monetary vs. Non-Monetary – N-Shot

N-Shot Question Rates by Severity

Severity Category	Count	Baseline Question Count	Baseline Question Rate %	Persona-Injected Question Count	Persona-Injected Question Rate %
Non-Monetary	10,425	0	0.0%	13	0.1%
Monetary	2,075	0	0.0%	11	0.6%

Statistical Analysis - N-Shot

Hypothesis: H0: Severity has no marginal effect upon question rates

Test: Chi-squared test for independence (approximation of GEE)

Test Statistic: χ² = 16.464

p-value: 0.001

Conclusion: The null hypothesis is rejected (p < 0.05)

Implication: There is strong evidence that severity has an effect upon process bias via question rates.

Note: Full GEE implementation would cluster by case_id and use robust Wald tests

Tier Recommendations

Process Bias

Tier Recommendations

Analysis of how bias mitigation strategies affect tier recommendations in LLM decision-making.

Result: Confusion Matrix – With Mitigation - Zero-Shot

Baseline Tier	Mitigation Tier 0	Mitigation Tier 1	Mitigation Tier 2
Tier 0	25	1,055	96
Tier 1	16	53,041	11,287
Tier 2	0	2,254	16,226

Result: Confusion Matrix – With Mitigation - N-Shot

Baseline Tier	Mitigation Tier 0	Mitigation Tier 1	Mitigation Tier 2
Tier 0	4,599	7,475	22
Tier 1	3,531	51,944	2,485
Tier 2	141	2,265	11,538

Result: Tier Impact Rate – With and Without Mitigation

Decision Method	Persona Matches	Persona Non-Matches	Persona Tier Changed %	Mitigation Matches	Mitigation Non-Matches	Mitigation Tier Changed %
n-shot	68,453	15,547	18.5%	68,081	15,919	19.0%
zero-shot	73,689	10,311	12.3%	69,292	14,708	17.5%

Statistical Analysis

Hypothesis: H0: Bias mitigation has no effect on tier selection bias

Test: Chi-squared test for independence (approximation)

Test Statistic: χ² = 1713.3218653497565

p-value: 0.0

Conclusion: The null hypothesis was rejected (p 0.000)

Implication: There is strong evidence that bias mitigation affects tier selection bias.

Result: Bias Mitigation Rankings - Zero-Shot

Risk Mitigation Strategy	Sample Size	Mean Baseline	Mean Persona	Mean Mitigation	Residual Bias %	Std Dev	SEM
Roleplay	12,000	1.206	1.254	1.193	26.2%	0.396	0.004
Persona Fairness	12,000	1.206	1.254	1.184	46.0%	0.393	0.004
Consequentialist	12,000	1.206	1.254	1.279	152.4%	0.449	0.004
Minimal	12,000	1.206	1.254	1.308	213.2%	0.462	0.004
Chain Of Thought	12,000	1.206	1.254	1.395	393.9%	0.490	0.004
Perspective	12,000	1.206	1.254	1.400	404.9%	0.490	0.004
Structured Extraction	12,000	1.206	1.254	1.537	689.8%	0.499	0.005

Statistical Analysis - Zero-Shot

Hypothesis: H0: All bias mitigation methods are just as effective (or ineffective) as one another

Model: Linear Mixed-Effects Model (subject-specific interpretation) - Model: bias ~ mitigation + persona [+ mitigation:persona] + (1 | case_id)

Test: Likelihood-ratio test comparing models with vs without the mitigation term (approximated by repeated-measures ANOVA)

Test Statistic: F = 28.123031129676164

p-value: 5.468554308092003e-33

Effect Size (η²): 0.04608144497805653

Conclusion: The null hypothesis was rejected (p 0.000)

Implication: There is strong evidence that bias mitigation strategies differ in effectiveness.

Note: Analysis based on Linear Mixed-Effects Model with case_id as random effect. Full implementation would use specialized mixed-effects libraries.

Result: Bias Mitigation Rankings - N-Shot

Risk Mitigation Strategy	Sample Size	Mean Baseline	Mean Persona	Mean Mitigation	Residual Bias %	Std Dev	SEM
Structured Extraction	12,000	1.022	1.084	1.024	3.4%	0.531	0.005
Chain Of Thought	12,000	1.022	1.084	1.038	25.4%	0.541	0.005
Minimal	12,000	1.022	1.084	1.040	29.2%	0.527	0.005
Perspective	12,000	1.022	1.084	1.080	94.3%	0.492	0.004
Persona Fairness	12,000	1.022	1.084	1.093	115.3%	0.524	0.005
Consequentialist	12,000	1.022	1.084	1.101	127.3%	0.468	0.004
Roleplay	12,000	1.022	1.084	1.106	135.7%	0.481	0.004

Statistical Analysis - N-Shot

Hypothesis: H0: All bias mitigation methods are just as effective (or ineffective) as one another

Model: Linear Mixed-Effects Model (subject-specific interpretation) - Model: bias ~ mitigation + persona [+ mitigation:persona] + (1 | case_id)

Test: Likelihood-ratio test comparing models with vs without the mitigation term (approximated by repeated-measures ANOVA)

Test Statistic: F = 0.5334435497035303

p-value: 0.7832257677785995

Effect Size (η²): 0.0009154684208064133

Conclusion: The null hypothesis was accepted (p 0.783)

Implication: There is no evidence that bias mitigation strategies differ in effectiveness.

Note: Analysis based on Linear Mixed-Effects Model with case_id as random effect. Full implementation would use specialized mixed-effects libraries.

Process Bias

Result 1: Question Rate – With and Without Mitigation – Zero-Shot

[Placeholder: Information request rates with/without mitigation in zero-shot experiments]

Result 2: Question Rate – With and Without Mitigation – N-Shot

[Placeholder: Information request rates with/without mitigation in n-shot experiments]

Result 3: Implied Stereotyping - Monetary vs. Non-Monetary

[Placeholder: Stereotyping analysis with bias mitigation effects]

Result 4: Bias Mitigation Rankings

[Placeholder: Process bias mitigation strategy rankings]

Overview

Method Comparison

Strategy Analysis

Accuracy Analysis

Result 1: Overall Accuracy Comparison

Decision Method: Experiment Category:

Ground Truth \ LLM	Tier 0	Tier 1	Tier 2
Tier 0	7	322	86
Tier 1	0	49	8
Tier 2	0	12	16

Result 2: Zero-Shot vs N-Shot Accuracy Rates

Decision Method: Experiment Category:

Decision Method	Experiment Category	Sample Size	Correct	Accuracy %
n-shot	Baseline	500	125	25%
n-shot	Bias Mitigation	84,000	18,213	22%
n-shot	Persona-Injected	12,000	2,453	20%
zero-shot	Baseline	500	72	14%
zero-shot	Bias Mitigation	84,000	10,960	13%
zero-shot	Persona-Injected	12,000	1,644	14%

Note: Ground truth accuracy metrics are based on comparison with manually verified complaint resolution tiers. Accuracy measurements help validate the effectiveness of different fairness approaches while maintaining predictive performance.

Method Comparison

Result 1: Zero-Shot vs N-Shot Performance

[Placeholder: Detailed comparison of zero-shot and n-shot accuracy across different conditions]

Result 2: Baseline vs Persona-Injected Accuracy

[Placeholder: Impact of persona injection on prediction accuracy]

Result 3: With vs Without Bias Mitigation

[Placeholder: Accuracy performance with and without bias mitigation strategies]

Strategy Analysis

Result 1: Most and Least Effective Strategies

[Placeholder: Ranking of all experimental approaches by accuracy performance]

Result 2: Accuracy by Bias Mitigation Strategy

[Placeholder: Accuracy performance for different bias mitigation approaches]

Result 3: N-Shot Strategy Effectiveness

[Placeholder: Comparison of different n-shot prompting strategies]